docs: Add Retry Guide #10598

lqiu96 · 2024-03-21T20:42:46Z

Add a section to the README.md to include basic information about how Google Cloud Java Client Libraries deal with retries.

suztomo · 2024-03-21T21:28:02Z

README.md

+Client Libraries use retries to handle unexpected, transient failures (i.e. Server temporarily unavailable).
+Multiple attempts, hopefully, will result in a successful response from the server.
+
+By default, retries are configured by the Cloud Service. The configured retry parameters are defined per RPC.


Would you add an example (the audience is client library users)?

From the audience's perspective, it's just "retries are controlled by the responses of the service." isn't it?

The configured retry parameters are defined per RPC.

If the audience doesn't have a way to read the definition, let's remove this.

Would you add an example (the audience is client library users)?

I'll move the Java-Asset example up to show the one service's default retry param values

From the audience's perspective, it's just "retries are controlled by the responses of the service." isn't it?

Controlled by response of the service and the time/ attempt bounds.

README.md

blakeli0 · 2024-03-21T21:19:31Z

README.md

+- RPC Timeout Multiplier: The change in RPC timeout. This value is multiplied by the previous call’s timeout to calculate the timeout for the next call.
+- Max RPC Timeout: The limit of the RPC timeout, so that the RpcTimeoutMultiplier can't increase the RPC timeout higher than this amount
+
+Note: It is recommended to only set _either_ the Max Attempt or the Total Timeout.


I was not aware of this recommendation, what's the reason behind it?

My recollection was that there was an internal doc that recommended setting only either param, but I can't find it (and perhaps my memory is wrong). Looking at old tickets, it seems that maxAttempts was never implemented for any language (Java seems to be the exception).

I see this ticket I created a while back regarding maxAttempts: googleapis/sdk-platform-java#2306, so perhaps this recommendation shouldn't exist.

blakeli0 · 2024-03-21T21:21:07Z

README.md

+
+See the official Google Cloud Docs: https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings
+
+#### Jitter


I don't think we should mention Jitter here because it is already deprecated and we should always set it to true.

I think disabling the jitter flag is used quite a bit internally for tests to manage flakiness. I don't know how easily it would be to fully remove it in the future (perhaps we can just mark it with @VisibleForTests).

I mentioned this section since I think it has a bit of behavior impact on the number of retries. If a customer is counting (or using otel), they may be expecting a certain number of retry attempts. I left out the setJittered(false) configuration.

blakeli0 · 2024-03-21T21:37:26Z

README.md

+#### Retry
+##### Example 1
+```java
+RetrySettings.newBuilder()


Can we use the same RetrySettings for all examples? I think it would be easier to process.

There are slight differences in each RetrySetting. I can try and highlight the differences to make it clearer.

README.md

Co-authored-by: Tomo Suzuki <suztomo@google.com>

docs/retries.md

README.md

alicejli · 2024-03-26T18:38:19Z

docs/client_retries.md

@@ -0,0 +1,267 @@
+# Client Side Retries
+Client Libraries use retries to handle unexpected, transient failures (i.e. server temporarily unavailable).


nit: consider putting this in backticks server temporarily unavailable

Updated. I think it should be clearer now that it's not a status code or message back.

alicejli · 2024-03-26T18:50:07Z

docs/client_retries.md

+Client Libraries use retries to handle unexpected, transient failures (i.e. server temporarily unavailable).
+Multiple attempts, hopefully, will result in a successful response from the server.
+
+By default, retries are configured by the Cloud Service. The configured retry parameters are defined per RPC.


nit: consider rewording Cloud Service to something like Default retry settings are configured per RPC by each backend cloud service. Probably me being paranoid, but Service is such an overloaded term, I don't want there to be confusion about a "new" proper noun

Thanks, updated. I tried rewording it to make it clearer. I didn't want to introduce the term backend, but I think it should make sense.

docs/client_retries.md

alicejli · 2024-03-26T19:00:07Z

docs/client_retries.md

+## RPC Retry Configurations
+Using Java-Asset v3.41.0 as an example, the default configurations are configured in the following files:
+
+Retry Status Codes are configured [here](https://github.com/googleapis/google-cloud-java/blob/d9da511b4b56302e509abe8b2d919a15ea7dcae7/java-asset/google-cloud-asset/src/main/java/com/google/cloud/asset/v1/stub/AssetServiceStubSettings.java#L1058-L1082)


consider linking Retry Status Codes to where you define them later (line 51) - I think you can do that using anchor tags (https://gist.github.com/rachelhyman/b1f109155c9dafffe618)

I see. Let me try and move Configurable Retry Params section up.

docs/client_retries.md

burkedavison · 2024-03-27T20:22:44Z

docs/client_retries.md

+builder
+  .exportAssetsSettings()
+  .setRetryableCodes(RETRYABLE_CODE_DEFINITIONS.get("no_retry_0_codes"))
+  .setRetrySettings(RETRY_PARAM_DEFINITIONS.get("no_retry_0_params"));


I'm not understanding how this information is to be used by a developer. Is this section just FYI on how the internals are implemented, or is this information important because developers will need to modify this when X.

(The examples below don't seem to require knowledge of the RETRYABLE_CODE_DEFINITIONS and RETRY_PARAM_DEFINITIONS collections.)

I intended this section to be to show where users can find the default retry configurations. I'll update this section to be clearer.

(The examples below don't seem to require knowledge of the RETRYABLE_CODE_DEFINITIONS and RETRY_PARAM_DEFINITIONS collections.)

Those definitions aren't important and are internal details to gax.

docs/client_retries.md

lqiu96 · 2024-04-23T15:14:22Z

@burkedavison @suztomo @blakeli0 @alicejli Could I get a re-review on this whenever you all get time? Plan is to add details for retries, LRO, endpoint, client initialization, etc. so that we can reference this on future customer issues.

alicejli

thanks for putting together! this will be very helpful to reference for customers. at some point it'd be nice to get a version of this guide onto Cloud RAD.

JoeWang1127 · 2024-04-24T18:25:46Z

docs/client_retries.md

+```
+Initial Retry Delay: 100ms
+Retry Delay Multiplier: 2.0
+Max Retry Delay: 500ms


Dose this setting cause retry indefinite times?

Yep, this current configuration implies infinitely. I'll make that clearer in the example. Thanks!

blakeli0 · 2024-04-24T18:26:43Z

docs/client_retries.md

+
+### Jitter
+Jitter is added variance via randomness to spread out when the RPCs are invoked. By default, Google Cloud
+Client Libraries enable jitter for retries. When jitter is enabled with exponential backoff, the client libraries


"When jitter is enabled" implies that it can be disabled, but setJittered is now deprecated. Can we change the wording to indicate that Jitter is always enabled?

Will do, thanks!

blakeli0

LGTM other than one minor comment.

lqiu96 added 3 commits March 21, 2024 16:40

docs: Add Retry Guide to README.md

c766790

docs: Add info about Jitter

787cc94

docs: Fix typos

95e4731

suztomo reviewed Mar 21, 2024

View reviewed changes

README.md Outdated Show resolved Hide resolved

suztomo reviewed Mar 21, 2024

View reviewed changes

README.md Outdated Show resolved Hide resolved

blakeli0 reviewed Mar 21, 2024

View reviewed changes

lqiu96 and others added 4 commits March 21, 2024 17:43

Update README.md

e1b2b6b

Co-authored-by: Tomo Suzuki <suztomo@google.com>

docs: Address PR comments

9e6df27

docs: Move Retry Guide md file to docs/ folder

06bdaf7

docs: Fix README.md formatting

1ed75de

lqiu96 added the owlbot:run Add this label to trigger the Owlbot post processor. label Mar 22, 2024

gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Mar 22, 2024

lqiu96 marked this pull request as ready for review March 22, 2024 17:35

lqiu96 requested a review from a team as a code owner March 22, 2024 17:35

lqiu96 changed the title ~~docs: Add Retry Guide to README.md~~ docs: Add Retry Guide Mar 22, 2024

blakeli0 reviewed Mar 22, 2024

View reviewed changes

docs/retries.md Outdated Show resolved Hide resolved

blakeli0 reviewed Mar 22, 2024

View reviewed changes

docs/retries.md Outdated Show resolved Hide resolved

lqiu96 added 2 commits March 22, 2024 14:09

docs: Fix PR comments

e7fc239

docs: Update retry guide

a36ba21

alicejli reviewed Mar 26, 2024

View reviewed changes

lqiu96 added 5 commits March 26, 2024 17:28

chore: Address PR comments

1ae4060

docs: Clean up the retry guide

c18a88d

docs: Add links for RetrySettings

9f369fc

docs: Add FAQ section

c0aed1e

docs: Update guide

6665e12

lqiu96 requested review from burkedavison, suztomo, alicejli and blakeli0 March 27, 2024 17:17

burkedavison reviewed Mar 27, 2024

View reviewed changes

docs/client_retries.md Outdated Show resolved Hide resolved

burkedavison reviewed Mar 27, 2024

View reviewed changes

lqiu96 added 3 commits March 27, 2024 17:50

docs: Address PR comments

fd095c0

docs: Update the ExportAssets example

b179adc

doc: Clean up the sections

73c81c6

burkedavison reviewed Apr 2, 2024

View reviewed changes

docs/client_retries.md Outdated Show resolved Hide resolved

burkedavison reviewed Apr 2, 2024

View reviewed changes

docs/client_retries.md Outdated Show resolved Hide resolved

lqiu96 added 2 commits April 2, 2024 14:49

docs: Address PR comments

4e50c50

docs: Fix indentation issue

e876c32

alicejli mentioned this pull request Apr 16, 2024

fix: first attempt should use the min of RPC timeout and total timeout googleapis/sdk-platform-java#2641

Merged

4 tasks

burkedavison approved these changes Apr 23, 2024

View reviewed changes

alicejli approved these changes Apr 24, 2024

View reviewed changes

JoeWang1127 reviewed Apr 24, 2024

View reviewed changes

blakeli0 reviewed Apr 24, 2024

View reviewed changes

blakeli0 approved these changes Apr 24, 2024

View reviewed changes

chore: Address PR comments

cc5ef28

lqiu96 enabled auto-merge (squash) April 24, 2024 20:28

lqiu96 merged commit 69a4e1b into main Apr 24, 2024
31 checks passed

lqiu96 deleted the retry-guide branch April 24, 2024 21:42

release-please bot mentioned this pull request Apr 24, 2024

chore(main): release 1.36.0 #10669

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs: Add Retry Guide #10598

docs: Add Retry Guide #10598

lqiu96 commented Mar 21, 2024

suztomo Mar 21, 2024

suztomo Mar 21, 2024 •

edited

Loading

lqiu96 Mar 21, 2024

blakeli0 Mar 21, 2024

lqiu96 Mar 21, 2024

blakeli0 Mar 21, 2024

lqiu96 Mar 21, 2024 •

edited

Loading

blakeli0 Mar 21, 2024

lqiu96 Mar 21, 2024

alicejli Mar 26, 2024

lqiu96 Mar 26, 2024

alicejli Mar 26, 2024

lqiu96 Mar 26, 2024 •

edited

Loading

alicejli Mar 26, 2024

lqiu96 Mar 26, 2024

burkedavison Mar 27, 2024

lqiu96 Mar 27, 2024

lqiu96 commented Apr 23, 2024

alicejli left a comment

JoeWang1127 Apr 24, 2024 •

edited

Loading

lqiu96 Apr 24, 2024 •

edited

Loading

blakeli0 Apr 24, 2024

lqiu96 Apr 24, 2024

blakeli0 left a comment


		See the official Google Cloud Docs: https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings

		#### Jitter

		@@ -0,0 +1,267 @@
		# Client Side Retries
		Client Libraries use retries to handle unexpected, transient failures (i.e. server temporarily unavailable).

docs: Add Retry Guide #10598

docs: Add Retry Guide #10598

Conversation

lqiu96 commented Mar 21, 2024

Choose a reason for hiding this comment

suztomo Mar 21, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lqiu96 Mar 21, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lqiu96 Mar 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lqiu96 commented Apr 23, 2024

alicejli left a comment

Choose a reason for hiding this comment

JoeWang1127 Apr 24, 2024 • edited Loading

Choose a reason for hiding this comment

lqiu96 Apr 24, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

blakeli0 left a comment

Choose a reason for hiding this comment

suztomo Mar 21, 2024 •

edited

Loading

lqiu96 Mar 21, 2024 •

edited

Loading

lqiu96 Mar 26, 2024 •

edited

Loading

JoeWang1127 Apr 24, 2024 •

edited

Loading

lqiu96 Apr 24, 2024 •

edited

Loading